Transformation-Grounded Image Generation Network for Novel 3D View Synthesis

摘要

We present a transformation-grounded image generation network for novel 3Dview synthesis from a single image. Instead of taking a 'blank slate' approach,we first explicitly infer the parts of the geometry visible both in the inputand novel views and then re-cast the remaining synthesis problem as imagecompletion. Specifically, we both predict a flow to move the pixels from theinput to the novel view along with a novel visibility map that helps deal withocculsion/disocculsion. Next, conditioned on those intermediate results, wehallucinate (infer) parts of the object invisible in the input image. Inaddition to the new network structure, training with a combination ofadversarial and perceptual loss results in a reduction in common artifacts ofnovel view synthesis such as distortions and holes, while successfullygenerating high frequency details and preserving visual aspects of the inputimage. We evaluate our approach on a wide range of synthetic and real examples.Both qualitative and quantitative results show our method achievessignificantly better results compared to existing methods.